feat(otelcol.connector): adding connector count #4913

hhertout · 2025-11-23T14:26:07Z

PR Description

The purpose of this pool request is to add the implementation of the connector.count of open telemetry in alloy.

The count connector generates metrics by counting telemetry signals (traces, logs, metrics) passing through the pipeline. This provides visibility into data ingestion volumes and allows tracking of data patterns across the collection layer.

OpenTelemetry connector count implementation: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/countconnector

Features

The count connector provides visibility into your telemetry pipeline:

Monitor ingestion volumes: Track the number of spans, logs, and metrics that Alloy processes
Add pipeline observability: Instrument the collector itself to detect bottlenecks or data loss
Filter and segment data: Count specific signals using OTTL conditions, for example HTTP requests or error logs
Group by attributes: Generate dimensional metrics grouped by service, environment, or other attributes

You can configure the count connector with custom metric definitions that use conditions to filter telemetry and attributes to group counts.

Default values are taken from the OpenTelemetry Collector repository to align with its default behavior. When you don't specify custom metrics, the connector generates standard count metrics for all telemetry types.

Example of alloy configuration:

otelcol.connector.count "default" {
  // Count only HTTP GET spans
  spans {
    name        = "http_get_requests"
    description = "Number of HTTP GET requests"
    conditions  = [
      "attributes[\"http.method\"] == \"GET\"",
    ]
  }

  // Count spans grouped by service and environment
  spans {
    name        = "spans_by_service"
    description = "Spans per service and environment"
    attributes {
      key           = "service.name"
      default_value = "unknown"
    }
    attributes {
      key           = "env"
      default_value = "production"
    }
  }

  // Count error and fatal logs only
  logs {
    name        = "error_logs"
    description = "Error and fatal log records"
    conditions  = [
      "severity_number >= 17",
    ]
  }

  // Count logs by environment
  logs {
    name        = "logs_by_env"
    description = "Log records per environment"
    attributes {
      key = "env"
    }
  }

  output {
    metrics = [otelcol.exporter.otlp.default.input]
  }
}

Default values are taken from the OpenTelemetry Collector repository to align with its default behavior.

Which issue(s) this PR fixes

Testing

Unit tests
Testdata validation for OTel→Alloy converter
Manual end-to-end testing with Prometheus export

Notes to the Reviewer

During development, I encountered a build failure in the vendored dependencies related to OpenTelemetry Collector's experimental profiles support. The error manifested in the debugexporter component.

It takes me a while to understand where the problem would be, but from my understanding:
The root cause was a version mismatch between OpenTelemetry Collector components. The project uses OpenTelemetry Collector v0.139.0, but the profiles API (go.opentelemetry.io/collector/pdata/pprofile) had been upgraded to v0.140.0 in the vendor directory. The debugexporter v0.139.0 expected the older profiles API, creating an incompatibility.

I didn't want to upgrade the entire OpenTelemetry Collector dependency tree to v0.140.x for several reasons:

It would introduce unrelated breaking changes
The scope exceeded the PR's goal of adding the count connector
It risked introducing additional compatibility issues with other Grafana Alloy components

As a workaround, I wrote integration tests to verify the behavior is correct.
I tested the complete behavior locally by fixing the vendor files, and running Alloy with the connector.count, to ensure everything runs correctly.

Impact: This workaround doesn't affect the count connector functionality. The profiles functionality in debugexporter isn't used by the count connector implementation.

PR Checklist

CHANGELOG.md updated
Documentation added
Tests updated
Config converters updated

kalleep · 2025-11-24T07:52:14Z

Hey, thanks for the pr. This is a duplicate of #4610 and #4550.

We have just not gotten around to review those

hhertout · 2025-11-24T08:07:11Z

Hello,

Sorry, I hadn't seen the second one you mentioned.
For the first one, after review it, my implementation is different, with tests, and the converter has also been implemented.

Let me know if you're still interested. Otherwise, I'll close it. There's no need for another duplicate if it's not useful.

kalleep · 2025-11-24T08:13:51Z

Yes I see that. I merged the pr to add support for these kind of connectors so you could rebase on that and remove your changes to connector.go.

There have not been any reactions to docs feeback in the other pr and I prefer the config structure added in this one so I am fine to go with this one instead.

go.mod

internal/component/otelcol/connector/count/count_integration_test.go

yann-soubeyrand · 2025-11-24T09:41:16Z

Hello, I’m the author of the other PR. I’m sorry, I didn’t have the time (and still don’t have for the moment) to move it forward. I chose this (somewhat weird) config structure, because it allows disabling the default counts (for example if you’re only interested in counting logs). I’m not sure it improves performances (I guess the unused counts do not run if there are no consumer for them, right?), so maybe it’s not worth it.

kalleep · 2025-11-24T09:48:02Z

Hello, I’m the author of the other PR. I’m sorry, I didn’t have the time (and still don’t have for the moment) to move it forward. I chose this (somewhat weird) config structure, because it allows disabling the default counts (for example if you’re only interested in counting logs). I’m not sure it improves performances (I guess the unused counts do not run if there are no consumer for them, right?), so maybe it’s not worth it.

Gotcha, yes it won't make any difference and that I why I prefer the config structure in this pr. I merged your changes to support these kind of connectors. So lets move ahead with this pr instead

hhertout · 2025-11-24T17:46:32Z

Hello, I’m the author of the other PR. I’m sorry, I didn’t have the time (and still don’t have for the moment) to move it forward. I chose this (somewhat weird) config structure, because it allows disabling the default counts (for example if you’re only interested in counting logs). I’m not sure it improves performances (I guess the unused counts do not run if there are no consumer for them, right?), so maybe it’s not worth it.

That makes sense. I can add that functionality if you think it would be valuable. It shouldn't be too complex...

hhertout · 2025-11-24T18:22:43Z

Yes I see that. I merged the pr to add support for these kind of connectors so you could rebase on that and remove your changes to connector.go.

There have not been any reactions to docs feeback in the other pr and I prefer the config structure added in this one so I am fine to go with this one instead.

I rebased, but I have to keep the if structure instead of the switch statements... due to this:

// ConnectorType() int implements connector.Arguments.
func (Arguments) ConnectorType() int {
	return connector.ConnectorLogsToMetrics | connector.ConnectorTracesToMetrics | connector.ConnectorMetricsToMetrics
}

As I understand, ConnectorType must return the type of connector used. In this case, it returns the three types of connectors, because it allows to generate metrics based on metrics, traces and logs inputs. This is slightly different from the span metrics for example, where there is only one type.

In the mentioned PR, the implementation have the following code:

func (args Arguments) ConnectorType() int {
	return connector.ConnectorLogsToMetrics
}

which is, from my point of view, not enough ? This will generate only metrics based on the log input right ?

Moreover, a switch statement doesn't work with combined flags since it checks for exact equality.
Here it works for both single-flag connectors (servicegraph, spanmetrics) and multi-flag connectors (count).

All existing connector tests pass with this approach.

WDYT ?

clayton-cornell

Some suggestions.

It's not critical (not blocking), but it'd be nice for consistency if the block sections were in the same order as given in the block table (after the suggested sorting is applied).

docs/sources/reference/components/otelcol/otelcol.connector.count.md

clayton-cornell · 2025-11-25T18:12:36Z

docs/sources/reference/components/otelcol/otelcol.connector.count.md

+# `otelcol.connector.count`
+
+`otelcol.connector.count` accepts spans, span events, metrics, data points, and log records from other `otelcol` components and generates metrics that count the received telemetry data.


Suggested change

# `otelcol.connector.count`

`otelcol.connector.count` accepts spans, span events, metrics, data points, and log records from other `otelcol` components and generates metrics that count the received telemetry data.

# `otelcol.connector.count`

{{< docs/shared lookup="stability/experimental.md" source="alloy" version="<ALLOY_VERSION>" >}}

`otelcol.connector.count` accepts spans, span events, metrics, data points, and log records from other `otelcol` components and generates metrics that count the received telemetry data.

If this is an experimental component, we need to add the shared element for experimental.

Thank you a lot for you review ! I've made the changes.

Though I've a question about this comment:

I don't know if I have to do it tbh... what's the default recommendation for this ? Should I mark it as experimental ?

There are a couple of things here we should verify.

Is this experimental? If yes, then we need the lable.stage in the metadata (already there) and the shared text block include (suggested). If no, then we need to remove or change the label.stage in the metadata and then skip this suggestion.

Is this a Community supported component? If yes, then we need to add the community label in the metadata, and add the shared community text block around line 12-13.

I'll defer to @grafana/grafana-alloy-maintainers for the decision on which way to go, and I can provide the doc suggestions to align.

Thank you for your answer !

docs/sources/reference/components/otelcol/otelcol.connector.count.md

kalleep · 2025-11-26T10:48:49Z

Yes you are right in that the other change to connector.go is not enough and while you change would work for count it would not work for other connectors if we decide to add them.

E.g. failoverconnector would not work without significant changes to this. This can connect to more than just metrics so it would fail this check.

So I think we should change that check a bit

if !outputsToMetrics(connectorType) && len(next.Metrics) > 0 {
	return errors.New("this connector cannot output metrics")
}

if !outputsToLogs(connectorType) && len(next.Logs) > 0 {
	return errors.New("this connector cannot output logs")
}

if !outputsToTraces(connectorType) && len(next.Traces) > 0 {
	return errors.New("this connector cannot output traces")
}

And add some helper functions

func outputsToMetrics(connectorType int) bool {
	return (connectorType&ConnectorTracesToMetrics) != 0 ||
		(connectorType&ConnectorMetricsToMetrics) != 0 ||
		(connectorType&ConnectorLogsToMetrics) != 0
}

func outputsToLogs(connectorType int) bool {
	return (connectorType&ConnectorTracesToLogs) != 0 ||
		(connectorType&ConnectorMetricsToLogs) != 0 ||
		(connectorType&ConnectorLogsToLogs) != 0
}

func outputsToTraces(connectorType int) bool {
	return (connectorType&ConnectorTracesToTraces) != 0 ||
		(connectorType&ConnectorMetricsToTraces) != 0 ||
		(connectorType&ConnectorLogsToTraces) != 0
}

For this to work we also need to update iota enum

const (
	ConnectorTracesToTraces = iota + 1 // thin is the change we need
	ConnectorTracesToMetrics
	ConnectorTracesToLogs
	ConnectorMetricsToTraces
	ConnectorMetricsToMetrics
	ConnectorMetricsToLogs
	ConnectorLogsToTraces
	ConnectorLogsToMetrics
	ConnectorLogsToLogs
)

And because we have not implemented any support for *ToLogs and *ToTraces yet we could return error that they are not supported yet.

…ks for all types of connectorType

hhertout · 2025-11-26T18:29:40Z

I also took this opportunity to add documentation on the helper functions about the bitwise checks.
Let me know if it's understandable and if you'd like any other modifications.

All tests are passing.

kalleep

Its fine to add comments but these a bit excessive, its fine to just describe what the function(s) is used for.

internal/component/otelcol/connector/connector.go

kalleep

Great, if you fix merge conflicts I can trigger ci

clayton-cornell · 2025-11-28T21:47:35Z

@kalleep Before you merge, there's an open question in the docs... is this experimental? Community? Either/or will require a tweak to the docs before we merge.

hhertout requested review from a team and clayton-cornell as code owners November 23, 2025 14:26

kalleep reviewed Nov 24, 2025

View reviewed changes

go.mod Outdated Show resolved Hide resolved

internal/component/otelcol/connector/count/count_integration_test.go Outdated Show resolved Hide resolved

feat(otelcol.connector): adding connector count

2710fe9

hhertout force-pushed the feature/otel-connector-count branch from 7154693 to 2710fe9 Compare November 24, 2025 17:56

clayton-cornell reviewed Nov 25, 2025

View reviewed changes

clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Nov 25, 2025

docs(otelcol.connector): update documentation after review

37b41d2

clayton-cornell reviewed Nov 25, 2025

View reviewed changes

docs/sources/reference/components/otelcol/otelcol.connector.count.md Outdated Show resolved Hide resolved

docs(otelcol.connector): fix documentation

a25e448

refactor(otelcol.connector): add helper function and add generic chec…

23b50c2

…ks for all types of connectorType

kalleep reviewed Nov 27, 2025

View reviewed changes

internal/component/otelcol/connector/connector.go Outdated Show resolved Hide resolved

internal/component/otelcol/connector/connector.go Outdated Show resolved Hide resolved

internal/component/otelcol/connector/connector.go Outdated Show resolved Hide resolved

docs(otelcol.connector): fix code documentation

93511db

kalleep reviewed Nov 28, 2025

View reviewed changes

hhertout and others added 2 commits November 28, 2025 17:50

Merge branch 'main' into feature/otel-connector-count

738cdc9

docs(otelcol.connector): fix changelog

24ea3f2

		# `otelcol.connector.count`

		`otelcol.connector.count` accepts spans, span events, metrics, data points, and log records from other `otelcol` components and generates metrics that count the received telemetry data.

feat(otelcol.connector): adding connector count #4913

Are you sure you want to change the base?

feat(otelcol.connector): adding connector count #4913

Conversation

hhertout commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Description

Features

Which issue(s) this PR fixes

Testing

Notes to the Reviewer

PR Checklist

Uh oh!

kalleep commented Nov 24, 2025

Uh oh!

hhertout commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kalleep commented Nov 24, 2025

Uh oh!

Uh oh!

Uh oh!

yann-soubeyrand commented Nov 24, 2025

Uh oh!

kalleep commented Nov 24, 2025

Uh oh!

hhertout commented Nov 24, 2025

Uh oh!

hhertout commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clayton-cornell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

clayton-cornell Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

hhertout Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clayton-cornell Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

hhertout Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kalleep commented Nov 26, 2025

Uh oh!

hhertout commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kalleep left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kalleep left a comment

Choose a reason for hiding this comment

Uh oh!

clayton-cornell commented Nov 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

hhertout commented Nov 23, 2025 •

edited

Loading

hhertout commented Nov 24, 2025 •

edited

Loading

hhertout commented Nov 24, 2025 •

edited

Loading

hhertout Nov 25, 2025 •

edited

Loading

hhertout commented Nov 26, 2025 •

edited

Loading